A study published in the December 5 issue of the Journal of the American Medical Association (JAMA) raised important philosophical issues with regard to researching the effects of nutrients in humans, demonstrating the confusion that persists when new research contradicts earlier research.

According to Andrew Shao, Ph.D., vice president, scientific and regulatory affairs, Council for Responsible Nutrition, Washington, D.C., “The problem that we have for the scientific community in evaluating the effects of nutrients in people is that everyone—from scientists to journalists to consumers—wants conclusive answers, consequently we’re always looking at what the ‘study du jour’ tells us and trying to make it answer all questions. But the reality is science doesn’t always move forward—there is some back and forth—and while research may seem to contradict itself, that should not be interpreted to mean one type of study trumps another, particularly when different studies ask and answer different questions. Seemingly conflicting data can exist side by side, when one understands that not all studies are asking the same questions in the same populations.”

The JAMA review set out to evaluate the change over time in the quantity and content of citations for two highly cited observational studies that found major cardiovascular benefits associated with vitamin E supplementation and to understand how these benefits continue to be defended in literature, despite contradicting evidence from large RCTs (randomized controlled trials). In order to make the findings more generalized, the study also examined the citations for observational studies showing the protective effects of beta-carotene on cancer and estrogen on Alzheimer’s disease. The study authors concluded that there is “an apparent split of stance in the scientific literature” when it comes to these substances.

The researchers gathered and reviewed 172 articles on vitamin E and cardiovascular health, 16 articles on beta-carotene and cancer prevention and 47 articles on estrogen for dementia prevention, rating them as favorable, equivocal and unfavorable. The sampled articles were published in 1997, 2001 and 2005 (before, early and late after publication of refuting RCTs) related to vitamin E and cardiovascular health and in 2006, which referenced highly cited articles proposing benefits associated with beta-carotene and cancer.

Pertaining to vitamin E, the researchers found that, despite the large RCTs that received a great deal of media attention, more than half of the articles that cited the observational studies were favorable towards a beneficial effect of supplemental vitamin E. The study authors said, “Even among articles that cited the contradicting HOPE [RCT] trial, rather than the positive epidemiological studies, the majority in 2005 still could not conclude that vitamin E was ineffective.”

Dr. Shao says “this suggests that researchers interpret research differently, depending on their bias and expertise. For pure scientific purposes, here’s a valid hypothesis to test: conduct a trial on secondary prevention in heart patients with a lifetime of bad habits that likely contributed to their heart disease to determine if a nutrient might provide some benefit. But it’s not valid to conclude from the results of that study that the nutrient doesn’t work. We can’t expect a simple vitamin supplement to reverse heart disease. So if that doesn’t happen, we must interpret the results appropriately by placing the study in the proper context and acknowledge that the results don’t answer the question of whether supplemental amounts of vitamin E in a healthy population could have prevented heart disease had it been used consistently over time in combination with other antioxidants.”  

The findings from the JAMA literature review related to beta-carotene and cancer and estrogen and dementia were similar, with the researchers finding that more than 62 percent of the articles in each of these subsets were favorable.

Dr. Shao states, “The RCTs with negative results attempted to answer the question, ‘can a supplemental nutrient treat or reverse a disease or a lifetime of unhealthy habits in patients who are also taking prescription medications?’ The observational studies with positive results attempted to answer the question, ‘if we start with a mostly healthy population generally free of disease, can we identify various diet/nutrient and lifestyle factors that make them more or less prone to disease?’ These are very different questions, making the studies incompatible for direct comparison and demonstrating that one type of study can’t necessarily be used to refute the other. We firmly believe that RCTs should not be thought of as the only rigorous research approach. As the study authors point out, ‘when randomized and observational studies disagree, it is incorrect to assume that nonrandomized studies are always wrong.’ Rather, we should put studies into the appropriate context and evaluate the total body of evidence, which includes RCTs and observational studies, and other types of research. We hope that the publication of this analysis will encourage all of those within the scientific community to become more involved in this discussion.”